This invention relates to accurate and precise representation of color for still and moving images, particularly sequences of digitized color images, including digital motion pictures and digital video.
CIE 1931 Standard Colorimetric Observer
During the period 1928 through 1931, J. Guild and W. D. Wright in England developed the CIE 1931 XYZ and xyz colorimetric system. The fundamental elements of this color representation are spectral mapping curves called x_bar, y_bar, and z_bar. These functions are based upon a transformation from human vision color matching measurements using amounts of relatively pure red, green, and blue. The red, green, and blue amounts were determined in 2-degree color patch matching experiments using a mixture of amounts of these red, green, and blue primaries to match arbitrary colors (other than red, green, or blue). The functions of the amounts to match arbitrary colors are embodied in spectral functions r_bar, g_bar, and b_bar. These functions have negative amounts in certain wavelength regions. The x_bar, y_bar, and z_bar spectral color matching functions are a linear transformation of Guild's and Wrights independent r_bar, g_bar, and b_bar measurements, taking into account the fact that the wavelengths of the red, green, and blue primaries were different between Guild and Wright. The x_bar, y_bar, and z_bar functions have all positive spectral amounts, unlike r_bar, g_bar, and b_bar. For general reference, see Color Science, Concepts and Methods, Quantitative Data and Formulae, 2nd Edition 1982, by Gunter Wyszecki and Walter Stanley Stiles, John Wiley and Sons; Color Appearance Models, 2nd Edition 2005, by Mark Fairchild, John Wiley and Sons; and Fundamental Chromaticity Diagram with Psychological Axes—Part 1, CIE 170-1:2006 (International Commission on Illumination, Commission Internationale De L'Eclairage, CIE) ISBN 3 901 906 46 0.
A discussion of this original color model work can be found in Wszecki and Stiles (W&S) in section 3.3 and section 5.5.6. It is significant to note that this work pre-dated computers and spectral-radiometers (which directly measure spectra digitally). The data for x_bar, y_bar, and z_bar are specified with four decimal digits, and at 5 nm intervals between 380 nm and 780 nm (see W&S Table II (3.3.1), page 736). This precision and accuracy is actually much higher than was implied by the original color examinations performed by Guild and Wright. Although the x_bar, y_bar, and z_bar spectral matching function data can accurately represent a theoretical color observer, these data do not represent actual people, but rather an average over a group of people (7 people tested by Guild, and 10 people tested by Wright). A problem with this testing approach is that there are substantial variations between individuals, and even variations for a single individual over time. There is theoretically an infinite number of different visible spectra which can integrate with x_bar, y_bar, and z_bar to yield the same value of CIE 1931 X, Y, and Z. This transformation of different spectra to the same CIE XYZ color, and the degree to which a different perceived color also sometimes results, is known as “metamerism”. Colors with identical spectra are called “isomers”, and such colors will always be perceived by each viewer as identical. However, even though two spectrally-identical colors are perceived as identical to each other, it is not possible to know how the color will appear in the mind's visual field of various individuals. There is variation from one individual to another, since each person essentially has his or her own individual spectral sensing functions at any given time. There may also be differences between individuals in the neural processing in the path between the retina and the visual cortex, and the resulting perceived color image. Further, there is evidence that people who need to carefully specify and distinguish colors, such as painters, cinematographers, and interior designers, may learn to improve their ability to distinguish and interpret colors compared to people who have not been trained in careful color distinction. Moreover, it is not known the degree to which color vision is hereditary due to human DNA codes, the degree to which it is neurologically developmental during early childhood, and the degree to which it can be further developed subsequently.
The CIE 1931 y_bar curve, which is based upon photopic luminance, is known to be incorrect based upon more accurate measurements done subsequent to 1931. This is significant in the CIE 1931 XYZ color model, since it affects color accuracy and not just luminance perception. The original photopic luminance work was performed by Judd in the early 1920's, and Judd later made a substantial correction in 1950 (see W&S Section 5.7.2), which became the basis of the 1970 Vos version of the XYZ color model (W&S section 5.5.2).
CIE 1964 Supplementary Standard Colorimetric Observer
In 1964, partly due to Judd's correction of y_bar, and also due to the discovery that a 10-degree color patch matched differently from a 2-degree color patch, the CIE added the CIE 1964 10-degree Supplementary Standard Colorimetric Observer (W&S section 3.3.2). The difference in average spectral functions between 2-degrees and 10-degrees is primarily caused by the yellow tint over the “macula latea” portion of the fovea of the human eye (the macular yellow pigment). This structural difference in the eye means that smaller patches of color are more yellow-filtered than larger patches.
There are thus a number of x_bar, y_bar, z_bar spectral color matching functions representing human color vision, with significant differences between them. FIG. 1 is a graph showing several existing x_bar, y_bar, z_bar color matching functions: the CIE 1931 (2-degree) version, the Vos 1970 version, and the CIE 1964 10-degree version.
Currently, CIE 1931 XYZ (and associated x_bar, y_bar, z_bar) is prominent (e.g., in the ICC color standard) as an engine for device-independent colorimetry. RGB representations are also the norm in color work, along with other three-primary (also called “tri-chromatic”) color representations, such as YUV, YPrPb, YCrCb, YIQ, LAB, and LUV. All of these tri-chromatic representations are fundamentally specified using CIE 1931 xy chromaticity coordinates for the color primaries, and associated matrix and gamma transforms, or are specified directly in terms of a transformation from CIE 1931 XYZ. Thus, at the present time, every common color system is specified in terms of the CIE 1931 x_bar, y_bar, z_bar spectral mapping.
Human Vision
In addition to color differences between various photopic x_bar, y_bar, z_bar trichromatic color matching functions (shown in FIG. 1), other factors affect color sensing, such as low light levels in common color viewing environments (e.g., motion picture cinema theaters).
The human eye is composed of “rods” and “cones”. Cones operate in high light levels, and see in color. This is known as “photopic”. Rods operate in low light, and see without color. This is known as “scotopic”. Movie projection operates in “mesopic” vision, such that the lower brightnesses are scotopic and higher brightnesses are photopic. The scotopic-luminance center wavelength is broadband centering in cyan, and is thus substantially bluer than photopic luminance which centers in yellow-green. Wyszecki and Stiles discuss “Tetrachromatic” color matching functions in section 5.6.2 as a means of dealing with mesopic vision. There is also some evidence that a small percentage of women have two slightly different wavelengths of red cones (in addition to green and blue), and can therefore potentially distinguish colors more finely than people with a single type of red cones. These women are thus somewhat “tetrachromatic” at high brightness, and thus potentially have five spectral wavelengths active in mesopic vision. It is further common for several percent of the male population to be somewhat color deficient (degrees of color blindness). This usually takes the form of two high-brightness cone colors instead of three, but there are other forms of color deficiency as well. All of these inter-personal variations make the goal of person-independent color difficult (or perhaps impossible).
It is also worth noting that tiny pigment wafers exist within cones which are gradually bleached by light during each day. These pigment wafers are gradually absorbed at the base of each cone during sleep each night, while new pigmented wafers are created at the top of each cone on the retina. A lack of sleep, especially when combined with the pigment-bleaching effects of high light levels (such as being outdoors), can therefore directly affect color perception. It is also likely that a better understanding will also develop over time with respect to these effects, particularly the affect of sleep and lack of sleep on color perception. Also, given the nature of this pigment-wafer cycle, it is likely that there may be a difference in color perception in the morning versus the middle of the day versus the evening. This is particularly interesting in light of the common practice of seeing movies and television in the evening, sometimes watching until late, just prior to sleep.
CIE170-1:2006 Modified CIE Colorimetric Observer
The CIE 170-1:2006 document, published by the CIE in January of 2006, allows “cone fundamentals” to define color mapping functions as a function of both age and angle of view to yield a “Modified CIE Colorimetric Observer”. For example, FIG. 2 is a graph of variation of cone fundamentals in CIE 170-1:2006 as a function of viewing angle (1 deg, 2 deg, 4 deg, and 10 deg) for age 35 years, and FIG. 3 is a graph of variation of cone fundamentals in CIE 170-1:2006 as a function of age (20 yrs, 40 yrs, 60 yrs, and 80 yrs) for a 2 degree viewing angle. As shown in FIG. 2 and FIG. 3, the average cone fundamentals (called l for long, m for medium, and s for short) vary significantly with viewing angle and with age. Further, sensitivity in certain wavelengths changes much more than at other wavelengths. Note for example the substantial variations near 500 nm and near 600 nm. The relative proportions of l, m, and s fundamentals can be seen to show large variations at these and at other sensitive wavelengths.
To quote Section 1.2 of CIE 170-1:2006: “Since observers, even within the same age bracket, may differ, a fundamental observer must be a theoretical construct based on averages. Any “real observer” will be different from the “Modified CIE Colorimetric Observer”. The starting functions are average colour-matching functions from a large sample (about 50 observers). An important inter-observer source of variability is the polymorphism of the photopigments, showing up as small, probably hereditary, variations of a few nanometers in peak wavelength of the cone fundamentals. Additional parameters such as the lens optical density, or macular pigment optical density, are given as average figures.”
The variations become particularly significant in light of the human visual ability to perform color matches at substantially less than 1%. It can be seen that the affects of age and viewing angle result in color determination variations which are more than an order of magnitude larger than 1% at sensitive wavelengths. Thus, any system for presenting precise and accurate color must take into account age and viewing angle, as well as inter-observer variations.
Note also that the CIE 170-1:2006 Modified CIE Color Matching Functions are based upon the 1959 Stiles and Burch data, as shown in FIG. 3 (5.5.6), FIG. 4 (5.5.6), and FIG. 5 (5.5.6) of Wyszecki and Stiles. This data shows variations on the order of several percent (and sometimes significantly more) between the 49 observers. Similar variation plots (which are clearly approximate) are shown in FIG. 1 (5.5.6) for the CIE 1931 observers used by Guild and Wright (7 and 10 observers, respectively) used in 1928-1929.
It should further be recognized that visual science is still actively investigating many relevant issues, and that there will likely be significant further refinements and improvements in visual spectral sensing models in the future. Further, there is likely to be a gradual improvement in understanding with respect to inter-personal variations in color matching functions, as well as factors related to heredity, ethnicity, age, gender, and viewing conditions. There may perhaps even evolve a better understanding of cultural preferences and biases in color appearance, as well as factors affecting color distinction including whether accurate color perception is used in one's employment (such as the work of a cinematographer, who is highly trained in the creation, management, and distinction of color).
The Use of Logarithmic Printing Density from Film Negative
Heretofore, movie masters have been film negative (or film duplicates thereof), a digital representation of film negative (using printing density units), or a reduced-range digital representation such as a digital television distribution master. For digital representations, there may also be many intermediate versions, but none of these is considered a master. Usually, only the reduced-range digital version embodies the final intended appearance, but it does so at the sacrifice of colors and brightnesses which are out of range for the digital television or digital cinema representation (as determined during mastering).
In 1990, the present inventor presented a paper at the Society of Motion Picture and Television Engineers (SMPTE) conference (see “The Use of Logarithmic and Density Units for Pixels” by Gary Demos, SMPTE Journal, October 1991, pp 805-816), proposing the use of logarithmic and density units for pixels. Based upon this paper, and the present inventor's request to SMPTE for a standardization effort, the SMPTE DPX (Digital Picture Exchange) file format was created to represent film negative density for motion pictures. As part of the SMPTE specification, a density representation known as “Printing Density” is used, which is the density of the film negative as seen by printing film (positive film for projection). However, printing density is not defined according to any industry-wide standard, and is thus difficult to use accurately or precisely. The more common standardized densities, known as “Status M” for film negatives, and “Status A” for print film, provide moderate accuracy for use in film developing consistency, but are insufficient for full spectral characterization.
In addition, negative films vary in their spectral sensitivity functions with exposure and color. Cross-color terms, such as variations in the amount of red in red, versus the amount of red in white or yellow, are also significant in negative film. Some of these effects occur due to inherent film light sensitivity spectral functions, and some occur due to chemical processes and dye-layer interaction during developing, and some occur due to dyes interacting with light spectra during the making of print film or scanning in a digital scanner. A further set of issues arise with the exposure, developing, and projection of motion picture print film, which is exposed from film negative. This process becomes more complex when camera photography uses one film negative, which is developed and scanned by a digital printer, and then manipulated digitally and recorded to another film negative in a digital film recorder (which is then printed to print film for projection). Copy film negatives and positives, known as film intermediate elements (internegative and interpositive) are also needed to provide thousands of film print copies for movie theater distribution.
When projected, a film print's dye spectral density is concatenated with the spectrum of the light source (usually xenon) and the spectral reflectance of the screen, to produce the spectrum which reaches the eye of the movie viewers. See FIG. 4, which is a graph of projected film spectra for red, green, blue, yellow, cyan, magenta, and white. Print film uses cyan, magenta, and yellow film dye layers to modulate red, green, and blue, respectively. The spectral transmittance of the various densities of dyes do not vary linearly, but rather alter their spectra shapes as a function of amount of density in that and other dye layers.
In addition to all of these film-based issues, the digital information from scanned and processed film and digital camera input may also be directly digitally output to a digital projection release master (which usually also involves compression). Often this master contains a simulation of the film printing process, and its color peculiarities. See, for example, FIG. 5, which is a graph of typical red, green, and blue primaries for digital cinema projectors.
It should be noted that the greatest success in precise color matching has come from side-by-side projected or displayed color images, with human-controlled adjustment of color to create a match. One such approach to the color matching task system is described in the U.S. Patent Application No. 60/198,890 and Ser. No. 09/648,414, “Film and Video Bi-Directional Color Matching System and Method” in the name of David Ruhoff and the present inventor. In that application, two images were placed side by side (or one above the other) on the same Cathode Ray Tube (CRT) screen. One image came from a digital film scan (such as using “printing density” units), the other image came from an electronic or digital camera. Both images were taken of the same scene, and were presented on the same screen. The colors of the side from the film scan were given a print simulation, and were adjusted to achieve a match. Therefore, subject to screen uniformity, both images utilized identical spectra during the matching process. An inverse was created from the film image's adjustment and print simulation process, which was then applied to the electronic/digital image. The data from the inverse process, applied to the electronic/digital image, then matched the digital film scan. The digital film scan and the inverse processed data were both output to a digital film recorder, which then received identical RGB input values (in “printing density” units in this example). The recorded negative film could then be optically printed and projected. Since the RGB values from the electronic/digital camera inverse process were very near the RGB values of the original digital film scan as a result of the matching process, the electronic/digital camera image would match the film-scanned image of the same scene. The film recorder RGB values were output to a laser film recorder, which used identical wavelengths of laser red, green, and blue primaries to record the intermediate film negative. Accordingly, the printing color process which followed was the same whether the original was from digitally scanned film or from the inverted electronic/digital camera. Since the spectra of the film recorder was the same in both cases, and the spectra of film developing and printing were the same (if done at the same time on the same roll of film), the projected film print results were identical. Again, the spectra of the digital film scan, when recorded on the film recorder, and the inverted electronic/digital camera image, were identical. Thus, nowhere in this color matching process was a color matching function used, nor was it needed. Further, there was no attempt made to match the projected film with its color gamut, white point, gamma function, and spectrum, to the video image on the Cathode Ray Tube (CRT), with its color gamut, white point, gamma function, and spectrum. However, color matching functions, based upon a standardized colorimetric observer, are needed when identical colors are desired from differing spectra.
In U.S. patent application Ser. No. 11/225,665, “High Quality Wide-Range Multi-Layer Compression Coding System” by the present inventor, efficient compression coding technology is described which maintains a wide dynamic range, and which can preserve the original image information by ensuring that coding error is less than the noise floor of the image itself. The system and method of U.S. patent application Ser. No. 11/225,665 can efficiently (via compression) preserve an extended color gamut range using negative numbers (for example, in red, green, and blue channels), and numbers above 1.0 (the most common logical presentation maximum in a mastering room context) using internal floating point processing. Additional channels (more than three) can also be coded. Further, a small amount of bit-exact (lossless) compression is available in the OpenExr 16-bit floating point representation, although the TIFF-32 standard does not currently offer useful compression. Without compression, a high resolution movie master is usually impractically large (many terabytes), even using today's large digital storage capacities. U.S. Patent Application No. 60/758,490, “Efficient Bit-Exact Lossless Image Coding Residual System” by the present inventor describes how to modestly compress moving images while retaining bit-exact original pixel values.
Color Printing on Paper
Another application where there is a need for accurate and precise control of color is color printing on paper. Color printers for home computers are becoming ubiquitous. There is a wide variation in the colors being produced by various color printers, however. One interesting example concerns some models of color printers which use additional color primaries (via additional colors of inks). Normal color printing modulates red, green, and blue light by using cyan, magenta, and yellow inks (or dyes), respectively. Sometimes black ink is also used for efficiency and black quality, although black typically does not carry any color information. In addition, some printers also use red and green as well as low-saturation photo-magenta and photo-cyan inks (or dyes). Some printers also utilize deep blue or violet, as well as orange, deep red, and other colors. In a typical image with red, green, and blue color channels (in addition to yellow, cyan, and magenta), there is little or no information on how to manage amounts of these additional colors. Other specialty inks (or dyes) are also sometimes used including “day-glow” fluorescent colors, which often convert invisible ultra-violet light into visible hot-pink, hot-green, and hot-blue, often with spikes in the resulting spectra. Some printers have other specialty inks having silver or gold metallic appearance. Some papers utilize “optical brighteners”, which convert invisible ultra-violet light into increased whiteness, sometimes also with spectral spikes (like day-glow inks and dyes).
The current practice for device-independent color representations for digital cameras and color computer printers is to use the ICC color standard, which is based upon CIE 1931 x_bar, y_bar, z_bar spectral mapping (via CIE XYZ or CIE LAB). This provides only approximate device-independence, and provides no information about how to use additional ink colors (beyond cyan, magenta, and yellow). It should be noted that the mapping between trichromatic images and display or printing devices with more than three color primaries is non-unique. There are thus numerous possible mappings from three primaries to more than three. Further, no such predetermined mapping is likely to accurately and precisely recreate all of the intended colors.
Color printing on film requires the use of a light source for presentation, with the light source having its own spectral characteristics. Color printing on paper requires the paper to be illuminated by light, which light will have its own spectral properties. While the film or color paper print can attempt to reproduce color with a specific spectrum of light, it cannot adapt to other light spectra.
Another significant issue is that the perception of color, brightness, and contrast is greatly affected by the color and brightness of the surrounding environment when viewing a color print or a displayed or projected image. The film or color paper print can attempt to correctly reproduce perceived colors in a single anticipated viewing surround, but cannot be made surround-independent. Further, there is no mechanism which could provide for variations between viewers, or within a single viewer (such as one's gradual adaptation to changing color and brightness perception when going from bright daylight to a darker indoor room).
The goal of precise and accurate device-independent color will remain illusive as long as such systems (such as the ICC color standard) are primarily based upon CIE 1931 x_bar, y_bar, and z_bar. Further, observer variations, and variations in image brightness and in surround color and brightness conspire to further complicate the task of accurate and precise reproduction of all of the colors in an image.
Use of CIE 1931 Chromaticity in Specifying File RGB Primaries
In the “High Quality Wide Range Multi-Layer Image Compression Coding System” patent application by the present inventor, a compression system is described which can retain extended range image representations using a floating point numerical representation. Publicly-available floating point image formats such as TIFF-32 and OpenExr can be directly supported as inputs and/or outputs for compression or other processing. However, these systems either do not define the spectral mapping of primary colors, or else specify color primaries using CIE xy chromaticity coordinates (which spectrally map using CIE 1931 x_bar, y_bar, and z_bar). The color precision and accuracy of these formats is therefore limited by the inherent limits of CIE 1931 x_bar, y_bar, and z_bar spectral mappings.
Differences in Spectral Sensing Functions of Cameras with Respect to Color Matching Functions
Another significant aspect of common color imaging systems is the spectral sensitivity function of color electronic cameras (usually these are digital cameras). Most color cameras use red, green, and blue spectral sensitivities which differ significantly from human color vision. The red spectral sensitivity of cameras, in particular, usually peaks at a much longer wavelength (deeper color of red) than does human vision. This has the affect of increasing color saturation for some colors. However, it also has the affect that some of the colors seen by a digital camera will not correspond to colors seen by the human eye (and cannot be unambiguously transformed into colors as seen by the human eye).
Camera films (usually negative films in motion picture use) similarly have spectral sensing functions which cannot be mapped into human vision. Camera film spectral sensing functions for the color primary channels (usually RGB) further vary with film exposure level, with amounts of the other color primary channels, and with other photo-chemical factors.
It is currently common practice to alter the original scene colors using a reference digital display or projector. This process has long been called “color timing” for the process of balancing and adjusting colors within film prints made from film negatives for stills and movies.
Such alteration can also be regional (such as in photo-retouch programs), and may be applied as moving image region color alteration (sometimes called “power windows” or “secondary colors” by the terminology of telecine color correction systems). Often these color adjustments are applied “to taste” and thus bear no relationship to the accurate nor precise reproduction of colors.
It can easily be recognized that the emission spectrum of the color reference display or projector is unlikely to match the sensing spectra of digital cameras, which in turn is unlikely to match the way colors are seen by the human visual system. Further, home viewing and home paper prints operate using yet different spectra.
There currently exists no practice for usefully reconciling the various sensing and emission spectra, outside of systems such as the ICC color standard (and similar systems) which are fundamentally based on the CIE 1931 x_bar, y_bar, and z_bar system, and thus do not take into account any of the common spectral variations in a typical imaging process (from camera to display, projector, color paper print, color transparent film, or video screen).
It is also useful to note that television systems in past decades relied upon a relatively consistent spectra due to consistent use of Cathode Ray Tubes (CRT's), with relatively consistent emission spectra. Reference CRT color monitors were commonly used to set reference color for scenes. Since nearly all end-user presentation in past decades was also via some form of CRT, the use of CIE 1931 chromaticities to specify the spectral mapping of colorimetry and primaries was adequate to a modest degree (although many displays differed substantially from color and gamma-curve calibration specifications). However, at present, most displays are no longer CRT's, and have a wide variation in emission spectra at mastering and at final presentation. Common current computer displays and television displays use LCD's, UHP lamps (metal halide) with projection LCD or DLP modulators, and plasma panel displays, each having significantly different emission spectra. Thus, at present, many systems rely much more heavily on CIE 1931, within which the red, green, and blue chromaticities are specified, to perform the spectral transformations necessary for reasonable color representation.
Of greater significance to the present invention is the current practice of discarding spectral information (both sensing and emission, especially the emission spectra of mastering displays and projectors) as images are processed and distributed. This is true for digital movie masters (which are relatively recent), digital television, web-based computer color image presentation, and color photography and color paper printing for personal and professional use.
In the case of digitally-scanned film negative, it is common practice to discard the filmstock information, which would identify the spectral sensing functions (although these vary with film emulsion batch), and which would identify the dye transmission spectra (which similarly vary), as well as the inter-layer interaction and exposure versus color interactions. Further, the spectral sensing functions of the digital film negative scanner, and perhaps the spectrum of its light source, are also discarded. What is ubiquitously provided is the RGB data, with unspecified spectral properties. This “raw” color digital negative is then color “timed” on a reference color projector or display, the spectra of which are similarly discarded.
It is easily seen that accurate and precise control and reproduction of color has heretofore proven elusive. Thus, an on-going technical challenge is attempting to accurately and precisely specify color in a manner that is device-independent, person-independent, color-patch-size independent, and time-independent.